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Because of perceived advantages, simulations are increasingly being used for performance-based training. 

This is a review of computer simulation as a learning tool, and its application within the context of disaster 
preparedness. Concepts from established learning theory are examined and research verifiable by 
empirical data is referenced. To summarize, this review found, despite a lack of empirical evidence, there 
exists wide support for computer simulation as a preferred tool in disaster preparedness. 
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After the terrorist attack on the World Trade Center on September 11, 2001, questions were raised as to the 
adequacy of the overall preparedness that exists in this country to serious, unanticipated mass emergency events. 
Subsequent assessments of Federal, State, and Local emergency first response agencies revealed some serious 
deficiencies in preparedness. Of all jurisdictions responsible for disaster response, a thorough literature review 
(DeGraffenreid, 1999) revealed local preparedness to be among the weakest links in the emergency response chain. 
Now, four years post 9/11, Hurricane Katrina has again raised serious concern regarding the adequacy of this 
country’s Federal, State and Local mass emergency response protocols. In this regard, one concept is clearly 
evident; a fully coordinated state of emergency response preparedness that links the national, state and local levels 
will ultimately depend on demonstrated competency of 1 st responders at the local level. This necessitates relevant 
training, opportunity for practice of learned skills, coordinated mock drills/exercises for experience and a 
comprehensive evaluation of local responder competencies. These are aspects of Human Resource Development 
that link directly to and are within the purview of current HRD professionals. With this in mind, and because of 
certain perceived advantages, computer simulations are increasingly being used for performance -based training. 
This paper describes a current review of computer simulation instruction as to its effectiveness for performance- 
based learning specifically in disaster preparedness training. 

Problem Statement and Research Proposition 

There is evidence that authenticity in realism (termed fidelity) to the mass emergency experience may be integral to 
the instruction necessary to produce competency in emergency response personnel (Lebow, 1994). However, sole 
reliance on first-hand application of competent response within actual emergency events by classroom-trained 
responders is impractical; there is no guarantee of proficient performance in the field. Further, it could be deemed 
an irresponsible, if not unethical, expectation. To approximate the type of fidelity necessary to practice a contrived 
mass emergency event through a full-scale role-play or drill is also impractical; the costs are prohibitive in terms of 
time, money, and personnel. To employ a “ next best” instructional alternative whereby costs are maintained at a 
feasible limit without compromise to competency in readiness, computerized simulations are currently being used 
and are being widely advocated for future training (Doyle, 2005). 

There is ample literature describing the use of simulations in instruction and learning in several different 
settings (Smith, 1986; Tennyson, 1987; West, 1991). However, evidence attesting to an increased learning 
effectiveness with computer simulated instruction vs. conventional classroom strategies is sparse. Remarkably, with 
the exception of several “no significant difference” (NSD) findings, research on computer simulations has been 
absent significant empirical evidence of an improved learning outcome over conventional, classroom methods. This 
seems to be in stark contrast to what may be anticipated. Computer simulations offer the opportunity for “risk-free” 
exercise experience without the undue consequence of property damage or personal harm. Further, it could be 
anticipated that the opportunity for practice with simulations is greater, allowing learned skills to be reinforced. The 
reported NSD results with simulated instruction, therefore, pose a conundrum when considering the generally held 
belief in the importance of skills practice as an essential component in instructional systems design (Seels & Richey, 
1994). The need to gain insight into this unexpected dilemma, as well as the increase in demand for its use, has 
warranted this review. With this in mind, the following proposition has been formulated. 
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It is proposed that, in lieu of experience gained in actual disaster incidents, simulations offer an equivocal 
option for effective skills practice in a mass emergency/disaster context within realistically represented learning 
situations at a significantly low risk of personal injury/property damage, and at manageable cost. 

To test the proposition, this review considered two research questions: 

1. Does computer simulation provide an effective instructional tool for disaster preparedness in terms of 
measured improvement in learning? 

2. As a learning strategy for disaster preparedness, does computer simulation provide for the desired outcome 
of improved performance in the field? 

Methodology & Scope 

This literature review focused on the effectiveness of learning outcomes that can be achieved through training or 
instruction within computer-based environments. Recognized learning theory (in particular, constructivist learning 
theory ), has been assessed where its application appears to support expected learning outcomes. Where possible, 
however, emphasis was placed on research that yields outcomes that are verifiable by empirical data. In this 
computer-based search, there were multiple key words used. It is notable that the most productive key phrase was 
found to be “computer simulation,” followed by a wide selection of specific fields dealing with learning and/or 
performance, (e.g., “computer simulation & experiential learning”, “computer simulation & performance outcomes”, 
etc.). With the exception of early studies considered to be seminal to the subject of learning theory (Bruner, 1966; 
Jonassen, 1977; Vygotsky, 1978), the review cites findings from recently published and/or currently recognized 
scholarly research, and takes into account the most recent applications of computer technology. In the course of 
this study, there has been an underlying salient question that may never be adequately answered, but none-the-less, 
needs to be raised. If there is an acceptable degree of preparedness, what is it and how is it best measured! While 
this study can provide direction towards a possible resolution to that question, it is not the intent nor objective of the 
study to provide a definitive answer. That scope is broader than this investigation can address. 

Theoretical Framework 

Simulation as an Instructional Strategy 

The instructional value of computer simulations by virtue of their richness of content, authenticity, and multi- 
sensorial affect, is seemingly unquestionable. A significant amount of the research reviewed seems to assume an 
inherent effectiveness of simulations as though intuitively obvious. The results suggest that simulations are at least 
as effective in learning outcomes as those achieved in actual laboratory exercises or case studies (Agrait, 2004; 
Gredler, 2004; Lee, 1999; Morgan, 2000; Smith, 1986; Tennyson, 1987; West, 1991; Yildiz, 1992). When one 
considers time and resource savings after development, and ever-increasing improvements in computer and 
instructional technologies, the use of simulations could become the primary instructional tool of choice for 
performance -based training. At the very least, their use could become standard as preferred instructional 
supplements for certain training situations. The considerable use of computer-based simulation in aviation pilot 
training is a familiar historical example (Franchi, 1995; Ericksson, 1999). It is worth noting that the physical 
infrastructure of training tools in that industry exceeds that which would likely be feasible for local emergency 
preparedness training. But the similarity in concept applies. 

In an interactive instructional computer simulation, participant(s) take on a particular role in actively 
addressing a given situation and experience the effects of their decisions, including the social and physical aspects of 
their role and their defined responsibilities and constraints. A simulation provides to participants an ill-structured 
problem (not strictly defined) of low transparency (without an immediately obvious solution) but with potentially 
several parameters (reinforcements or constraints) and possible pathways of action (options) that may appropriately 
address the problem (Jonassen, 1997). 

Despite claims of significant effectiveness, it has been difficult to locate sufficient empirical evidence to assess 
the actual positive learning and performance outcomes that are realized through simulation instruction. Without 
such data, there remains an inability to predict with certainty the degree of knowledge that will be transferred from 
simulated exercises to real, ongoing mass emergency/disaster events. 

Simulation Application in Disaster Preparedness 
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When a disaster incident occurs, the Fire, Police, Emergency Medical (EMS) and Public Flealth 1 agencies 
individually respond to employ the specific aid that each agency has been professionally trained to provide. 
Flowever, what are not always prepared for are the unique and unanticipated inter-agency interactions (variables) 
that may be encountered which may require unplanned coordination but are none-the-less necessary to effectively 
mitigate the emergency and the extent of damage (outcome). 

External and internal factors within each agency can influence the effectiveness of that agency’s response and, 
in turn, the effectiveness of the total, four-agency emergency response. Any inconsistency or conflict in 
coordination and/or collaboration amongst responding agencies will produce a performance gap between the optimal 
and actual actions taken. Minimization of any performance gap(s) is the ultimate objective of disaster preparedness. 
Figure 1 shows an example of an assessment model as it would apply to one of the response agencies. Public Flealth. 



Performance Assess nient Model 




External Internal -critical thinking 

Variables Variables 



Figure 1. Assessment Model for Public Health 

Like Public Flealth, the Fire, Police and Medical (EMS) agencies are also impacted by various external and 
internal factors that will influence effectiveness of overall performance (mitigation). As depicted in the model, 
optimal response in a simulated exercise is that performance defined as optimal by subject matter experts (SMEs) 
who are experienced in disaster response and who have been familiarized with the pertinent factors included in the 
simulated instructional exercise. All agencies need to interact collaboratively, and the actual agency(s) performance 
would be evaluated according to SME-defined criteria. 

Results and Findings 

Theories of Learning Associated with Computer Simulation 

No single learning theory accounts for the use and effectiveness of simulation in learning. Rather, several 
theories lend validity to its application. The fundamental similarities in these theories are that learning is an active 
process, best experienced within a realistic context, to allow for application and use (transfer) of knowledge to 
realistically the same, or similar, situations. All of these theories are, or are related to, the constructivist learning 
theory. Constructivism has been described (Kauchak & Eggen, 1998, p. 184). as a "view of learning in which 
learners use their own experiences to construct understandings that make sense to them, rather than having 
understanding delivered to them in already organized form. Learning activities based on constructivism put learners 
in the context of what they already know, and apply their understanding to authentic situations." Table 1 provides a 
summary of several learning approaches that have incorporated simulation as an instructional strategy. As a 



1 We realize that the first responder community can be broader than these four agencies to include State, Federal, and other local agencies and 
even broader to the private sector. However, these four are the primary local organizations designated as first responders, so we will concentrate 
our efforts initially on them. 
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detailed examination of these theories is not practical nor within the scope of this article, the interested reader is 
directed to the references cited to pursue additional elaboration. 

Variables Affecting Computer-Simulated Instruction Outcomes 

Degree of Simulation Fidelity & Richness. As this review earlier proposed, to be most effective as a learning 
tool, computer simulations need to provide learning experiences that are as representative of the actual experience as 
possible (Standen, 1996). This is the simulation fidelity and is characterized by both a realistic experience and a 
realistic learning environment. Simulations low in fidelity may be low in realism and presumably lower in learning 
effectiveness. The Simulation Interoperability Standards Organization (SISO) adopted a formal definition of 
simulation fidelity as “the degree to which a model or simulation reproduces the state and behavior of a real world 
object, or the perception of a real world object, feature, condition, or chosen standard in a measurable or perceivable 
manner.” 



Table 1. Learning Theories and Simulation Strategies 
Theories Principles 

Experiential Learning Learning is a cycle involving 
(Kolb. 1984) experiencing, reflecting, thinking, 

and acting. 



Situated Learning 
(Lave, 1990) 

Problem-Based 
Learning 
(Vygotsky, 1978) 
Discovery Learning 
( Bruner , 1966) 

Computer-Supported 

Collaborative 

Learning 

(Koschmann. 1996) 



Learning is a function of the 
activity, context, & culture in 
which it occurs. 

Focus is on problem to be solved 
vs. content to be mastered 

Learners explore problem and 
discover/retain solution concepts 
aided by prior knowledge 
Learning occurs through 
knowledge-sharing and 
collaboration among multiple 
participants. 



Goals 

Usable knowledge 
created through 
transformation of 
experience 

Application of learned 
knowledge in similar (not 
necessarily same) context 
Develop reasoning skills, 
self-directed strategies 

To independently solve 
problems through 
informed decisions 
Facilitate & optimize 
collaborative 
understanding through 
computer-assisted 
learning. 



Strategies 

Simulated exercise for experience. 
Role Plays. Coaching. 

Learning through Doing. 

Knowledge must be presented in an 
authentic context. It requires social 
interaction and collaboration. 

Present learners with “ill-structured” 
simulated problem situation to be 
mitigated. 

Use virtual simulated exercises, role 
plays, concept mapping. 

Computer simulations, games 
providing authentic learning 
scenarios for multiple participants. 



There is also indication that the amount of detail in a simulation has an impact. This degree of detail in a 
simulation is defined as its richness. Because any simulation is only a representation of some reality, most of the 
value is in its ability to simplify the complexity of the real world into a form that is comprehendible and usable. 
Frequently, simulation designers include all the fidelity and richness they can afford, without consideration of the 
overload burden that is created and the reduced benefit that results. A highly detailed and “over-engineered” 
training simulation may in its complexity obscure the real issues for which training is required. That would 
undermine one of the real values of simulations, i.e., the abstracting away of irrelevant details. Each component to 
be learned from a particular computer-simulated instruction can have its own learning objective (e.g., factual 
comprehension, procedural accuracy, etc.). Accordingly, the degree of fidelity and richness may be relevant to a 
specific learning context, and their significance will likely vary with the needs of that particular context. 

Qualitative vs. Quantitative Fidelity. A quantitative measure of simulation fidelity as described in the SISO 
definition is “objective” and not easily obtained. Qualitative measures may be “subjective” but, in general terms, 
they can be understood (e.g. “spicy” food, “chilly” evening, etc). Without resorting to various quantitative methods 
that may prove ambiguous, it could be possible to compare one simulation to other simulations designed to meet the 
same objectives in order to gauge its effectiveness for that purpose (Quarantelli, 2000). In doing so, it is 
conceivable that the validity of the simulation in meeting intended objectives may be determined. It may be that the 
validity of a simulation is more critical than its fidelity. Here, simulation validity refers to that quality of being 
inferred, deduced, or calculated correctly enough to specifically apply to a particular problem situation. 

Novice vs. Experienced Simulation Learners. Choice of simulation representations will depend not only on the 
application context, but on the level of participants’ experience with learning from simulations. Novices learn best 
with lower level fidelity, while experienced learners do better with high fidelity (Chen, 2004). The difference 
appears to be due to the richness of the simulated presentation. For novices, high fidelity may provide too much 
information to process rapidly, resulting in response delays. For learners experienced with simulation instruction, 
too simplistic a presentation may not engage the learner and concentration can drift or be lost. Where novices may 
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also be lacking in prior emergency response experience, this will further impact computer simulated instruction and 
delay. Acceptable individual competence may need to be gained before group (team) competence in response can 
be accomplished. 

Lessons from Computer-based Instruction. A recent meta-analysis (Lowe and Holton, 2005) of computer- 
based instruction sheds some light on the various interactions of instructional design, learner characteristics and 
learning outcomes that have relevance to this investigation. They investigate a number of relationships that are too 
numerous to review here, but there are important conclusions that should lend themselves to generalization for 
computer-based simulation instructional environments. Among those of importance include the following: 

• Learning to perform in complex problem-solving environments requires more elaborate feedback 
mechanisms and considerably more practice opportunity. These are especially effective with learners 
having high meta-cognitive skills. Where there is a high internal locus of control, high meta-cognitive 
skills, and high motivation, there is less need of support required to achieve effective learning. 

• Instructional strategy selection should precede visual design for learning to be most effective. Over- 
engineering graphic interfaces, including screen display, the amount of information available and the 
feedback process is likely to result in a mismatch between learning goals and instructional strategy. 

• Instructional strategy design is also influenced by certain learner characteristics such as self-directedness 
and motivation, prior computer experience, amount of external/environmental support (appropriate 
computer equipment, technical support, time released to participate, support from peers, supervisors, 
management) and clearly specified personal learning objectives. 

• Simulations would appear to be better suited for environments where higher-order learning outcomes are 
desired. In fact, Lowe and Holton found evidence that simple learning tasks requiring the use of simple 
resources led to learning outcomes that were insignificantly impacted by computer-based delivery. It 
appears the use of simulation for low level learning outcomes would be a waste of resources. In contrast, 
emphasis on higher-order outcomes would require learners to generate solutions to problems and 
demonstrate the application of rules learned. 

Why the Ambiguity in Performance Measurements? 

In reviewing the literature, and in discussions with key simulation researchers, there are at least two causes proposed 
which may explain the NSD effectiveness found between conventional classroom and computer simulation 
instruction. The first appears to be the ambiguity inherent in the measurement methods that have been used. Much 
of the evaluation has involved surveys or anecdotal recall and not quantitative measurement. A basic means of 
determining if transfer of learning has occurred is by direct observation of the learned tasks/procedures being 
applied (Kirkpatrick, 1998). In an emergency scenario, there may be several ways to address a single problem or 
procedure, complicating the observation of a “correct” learned application. Events occur simultaneously, by chance, 
and not necessarily in sequence, further hindering evaluation. It seems comprehensive evaluation would require an 
army of evaluators, each measuring a specific response that can be directly correlated back to a specific instructional 
aspect of the simulation, and that clearly is impractical. 

The second explanation relates to the ambiguity that exists in the computer-simulation learning process itself. 
The process still has not yet been satisfactorily defined to be clearly understood. It is here where side-by-side 
comparisons become somewhat futile, like comparing the proverbial apples to oranges. Simulation is multi-sensory 
in its instructional impact unlike any other instructional strategy. As such, a comparison assessment of simulation 
with another instructional method (e.g. conventional classroom) is inappropriate because the measurable outcomes 
are not likely to be the same. Two simulations, however, may be compared against each other (Quarantelli, 1989), 
and the inherent pros/cons of the simulation instruction technique itself should reasonably be identifiable, and 
measurable. A comparison of two like simulations would seemly provide the most meaningful measurable results. 

A Design Framework for Disaster Preparedness Instruction 

Disaster preparedness involves a “readiness to respond and suggests purposive, or anticipatory action. It implies the 
knowledge of appropriate behavior, conveys the need for training and practice, and subsequent capability to achieve 
purposes.” (Gillespie, 1993). This review has focused on that aspect of “readiness to respond” that computer 
simulated instruction may provide. There are nine (9) attributes that have been identified which can guide the 
development of exercises / drills / participatory events, etc., to produce an effective degree of disaster preparedness. 
Experiential Learning 
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The design of the exercise should, as a minimum, take into account primary principles of experiential learning, 
including situated and adult learning (Fenwick, 2000; Flenning, 1996). The experiential learning cycle of 
Experience Reflection Generalization-} Application (recycling back to)-> Experience is advocated. Experiential 

learning principles hold that an individual acquires useful knowledge or learning for future behavior and application 
to necessary tasks through the experiences of participation, as opposed to pure cognitive learning that may be 
acquired through structured classroom situations. Thus, a more participative, interactive, and reflective environment 
for learning should be designed to incorporate this approach to learning. It should be noted that such environments 
(while seeming to mirror a preferred approach advocated by many first responders) may not result in a single most 
effective learning outcome (for an individual or a group). 

Multi-Learner/Collaborative Activities 

The potential use of exercises for learning events which assumes a multi-learner environment is important. 
More specifically, groups of learners should be placed together to achieve the exercise objectives. This is one of the 
fundamental components of the Activity Learning Theory, whereby higher order/critical thinking learning is 
enhanced in individuals interacting within a group experience (Flannafin & Land, 1997). Careful attention is 
required regarding not only how the groups are formed (in order to best accomplish the objectives) but also 
designing activities that utilize sound participatory and collaborative principles. While such activities may not 
involve ‘teams’ per se, team-like activities that benefit individual learners is an important consideration. 

Definition of Objectives 

One of the issues emphasized in the literature, and personally observed in numerous exercises over a two-year 
period (2003-2005), is the specification of measurable objectives that are aligned with the exercise environment. 
Whether it be the lack of particular objectives, the over-specification of objectives, or the imprecise wording of 
objectives, more attention to the development of clear, attainable objectives is essential. Furthermore, the objectives 
should be stated with language that specifies performance (what participants are supposed to do in action terms), 
conditions (the situation, describing available resources), and criteria (how well the learner must perform, or what is 
considered acceptable). 

Realism (Fidelity) 

The situation posed by the exercise should have two components of realism (ThoughtLink, Inc., 2004). One is 
that the scenario should reflect as much as possible the general type of conditions present in a disaster or emergency, 
possibly based on past real occurrences. Second the exercise should be realistic on a more personal side, reflecting 
the regional environment with local landmarks, use of local resources and organizational culture considerations (i.e. 
with specificity). The degree of fidelity must take into account prior simulation learning-experience, with the 
complexity being low to moderate with less experienced individuals and increasing in detail with more experienced 
learners. 

Repetition (Frequency) 

It is impractical to assume that a single exercise focused on a given knowledge/skill set is sufficient to assure 
competency. Repeated experiences are needed to strengthen and instill the correct behaviors associated with 
necessary performance, increase the fluency in decision-making skills, and increase problem-solving capacity. 

Safely 

Exercises should reinforce the use of recognized safe protocol as a requirement for disaster/emergency response 
and/or performance under adverse conditions (ThoughtLink, Inc., 2004). Thus, exercises should expose participants 
to simulated (only) hazardous conditions for problem-solving and simulated (only) dangerous events which are also 
necessarily linked to the use of safe procedures and actions to be reinforced. Critical Thinking & Problem-Solving. 

When considering the use of advanced exercises (i.e., those that go beyond simple awareness and application 
of basic skills and knowledge), activities should be constructed with unusual or unanticipated consequences. The 
objective is to provide experience in modeling the control of the unexpected. Such activities would introduce high- 
stress response situations, but in the “risk-free” simulated format. The degree of accuracy, efficiency, and timeliness 
should be specifically defined in measurable terms. 

Use of Existing Protocols 

Exercises should be designed for participants to use practiced, standardized procedures in order to test both the 
adequacy/effectiveness of the procedures as well as the ability of participants to correctly and appropriately apply 
them. This can assist in potential refinement in the efficiency, effectiveness and clarity of the procedure 
documentation. In addition, sophistication in the exercise design could be modulated to generate data to be 
subsequently examined/analyzed by protocol designers for the purpose of performance improvement. As an 
extension, this activity would also have implications for organizational performance. 

Measurement of Performance 
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A critical component of learning exercises is that they be designed to facilitate the collection, measurement, and 
analysis of learner performance. Actions such as response time, decision steps, resource expenditures, 

communication, information sharing, use of expert advice, casualty counts, etc., can be recorded by observers during 
the course of the exercise. Results of such measurement could potentially be linked to certification procedures. 

Conclusion and Recommendation 

Few studies have been successful in generating quantitative empirical data which demonstrates specific 
improvements in measurable learning effectiveness of simulations or improved performance as a result of computer 
simulated preparedness instruction. Much of the research data is based on interview and observational data that 
lacks substantiation, or on determination of efficiencies in costs or time reduction. Yet, the use of computer 
simulations for instruction in a multitude of disciplines is increasing almost exponentially as technology advances 
are achieved and used. Quite literally, the use of computer simulation instruction appears to be out-pacing the 
research on its learning effectiveness. The experience in the use of computer simulation and simulators in such key 
instructional contexts as pilot/astronaut flight training and nuclear power plant operation provide a historical record 
in support of simulation effectiveness with regard to high stress situations. With this historical experience, and based 
on the body of literature reviewed, there has been a significantly favorable experience with the use if computer 
simulations in instruction in general, and now specifically, with instructional exercises for disaster preparedness. 
Based on the currently limited quantitative evidence, but more considerable qualitative support, the body of 
evidence from this literature review is supportive of simulation as an effective instructional strategy. Given the 
appropriateness of their fidelity and richness, simulations can provide realistic learning situations at the lowest 
possible risk of personal injury/property loss. With a reduced reliance on significant resources, simulations can 
provide an equivocal learning experience at manageable cost. 

In the emergency preparedness field of application, there is currently a significant interest in a “blended 
instruction” approach to first responder preparedness training (e.g. Department of Homeland Security. 2004). 
Blended Instruction (a.k.a. hybrid instruction) involves the use of a mix of conventional and online instructional 
strategies. Computer simulation is supported to contribute to a blended format for mass emergency/disaster 
preparedness training, particularly where performance vs. factual knowledge is the instructional criterion and 
objective for learning. The design strategy framework for simulated disaster preparedness should be subject to 
continuous improvement modification as technology progresses, experience is gained from its use, and its 
effectiveness is validated via ongoing evaluation. In linking this review to HRD, this article constitutes an 
examination and synopsis of the current research regarding computer simulation effectiveness as a learning tool, in 
this case, specifically to disaster preparedness. The information can arguably apply to any team-oriented training 
context where collaboration is critical to group effectiveness. From a Human Resource Development perspective, 
the knowledge gained from a practical and theoretical standpoint supports and furthers the group-think concept to 
achieving both individual and team competencies. In the case of disaster preparedness competency, this is critical 
where experience through physical, real-world practice is constrained yet competence in on-site performance is 
crucial. This responsibility to develop competency in personnel more often than not is a direct responsibility of the 
Human Resource Development professional. This review can assist the HRD professional in that effort and also 
supply valuable empirical data, which is currently lacking, but which would support continuing research on 
computer simulation effectiveness in general. 
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